Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics
Identifieur interne : 001811 ( Main/Exploration ); précédent : 001810; suivant : 001812Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics
Auteurs : Yann Mathet [France] ; Antoine Widlöcher [France] ; Karën Fort [France] ; Claire François [France] ; Olivier Galibert [France] ; Cyril Grouin [France] ; Juliette Kahn [France] ; Sophie Rosset [France] ; Pierre Zweigenbaum [France]Source :
English descriptors
Abstract
Computing inter-annotator agreement measures on a manually annotated corpus is necessary to evaluate the reliability of its annotation. However, the interpretation of the obtained results is recognized as highly arbitrary. We describe in this article a method and a tool that we developed which "shuffles" a reference annotation according to different error paradigms, thereby creating artificial annotations with controlled errors. Agreement measures are computed on these corpora, and the obtained results are used to model the behavior of these measures and understand their actual meaning.
Url:
Affiliations:
- France
- Basse-Normandie, Grand Est, Lorraine (région), Région Normandie
- Caen, Metz, Nancy
- Université de Caen Basse-Normandie, Université de Lorraine
Links toward previous steps (curation, corpus...)
- to stream Hal, to step Corpus: 003084
- to stream Hal, to step Curation: 003084
- to stream Hal, to step Checkpoint: 001423
- to stream Main, to step Merge: 001839
- to stream Main, to step Curation: 001811
Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics</title><author><name sortKey="Mathet, Yann" sort="Mathet, Yann" uniqKey="Mathet Y" first="Yann" last="Mathet">Yann Mathet</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-388300" status="VALID"><orgName>Equipe Hultech - Laboratoire GREYC - UMR6072</orgName><desc><address><country key="FR"></country></address></desc><listRelation><relation active="#struct-150" type="direct"></relation><relation name="UMR6072" active="#struct-441569" type="indirect"></relation><relation active="#struct-300358" type="indirect"></relation><relation active="#struct-300266" type="indirect"></relation></listRelation><tutelles><tutelle active="#struct-150" type="direct"><org type="laboratory" xml:id="struct-150" status="VALID"><orgName>Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen</orgName><orgName type="acronym">GREYC</orgName><desc><address><addrLine>Boulevard du Maréchal Juin - 14050 CAEN Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.greyc.fr</ref></desc><listRelation><relation name="UMR6072" active="#struct-441569" type="direct"></relation><relation active="#struct-300358" type="direct"></relation><relation active="#struct-300266" type="direct"></relation></listRelation></org></tutelle><tutelle name="UMR6072" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-300358" type="indirect"><org type="institution" xml:id="struct-300358" status="VALID"><orgName>Ecole Nationale Supérieure d'Ingénieurs de Caen</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300266" type="indirect"><org type="institution" xml:id="struct-300266" status="INCOMING"><orgName>Université de Caen Basse-Normandie</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Caen</settlement><region type="region" nuts="2">Région Normandie</region><region type="region" nuts="2">Basse-Normandie</region></placeName><orgName type="university">Université de Caen Basse-Normandie</orgName></affiliation></author><author><name sortKey="Widlocher, Antoine" sort="Widlocher, Antoine" uniqKey="Widlocher A" first="Antoine" last="Widlöcher">Antoine Widlöcher</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-388594" status="VALID"><orgName>Equipe CODAG - Laboratoire GREYC - UMR6072</orgName><desc><address><country key="FR"></country></address></desc><listRelation><relation active="#struct-150" type="direct"></relation><relation name="UMR6072" active="#struct-441569" type="indirect"></relation><relation active="#struct-300358" type="indirect"></relation><relation active="#struct-300266" type="indirect"></relation></listRelation><tutelles><tutelle active="#struct-150" type="direct"><org type="laboratory" xml:id="struct-150" status="VALID"><orgName>Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen</orgName><orgName type="acronym">GREYC</orgName><desc><address><addrLine>Boulevard du Maréchal Juin - 14050 CAEN Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.greyc.fr</ref></desc><listRelation><relation name="UMR6072" active="#struct-441569" type="direct"></relation><relation active="#struct-300358" type="direct"></relation><relation active="#struct-300266" type="direct"></relation></listRelation></org></tutelle><tutelle name="UMR6072" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-300358" type="indirect"><org type="institution" xml:id="struct-300358" status="VALID"><orgName>Ecole Nationale Supérieure d'Ingénieurs de Caen</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300266" type="indirect"><org type="institution" xml:id="struct-300266" status="INCOMING"><orgName>Université de Caen Basse-Normandie</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Caen</settlement><region type="region" nuts="2">Région Normandie</region><region type="region" nuts="2">Basse-Normandie</region></placeName><orgName type="university">Université de Caen Basse-Normandie</orgName></affiliation></author><author><name sortKey="Fort, Karen" sort="Fort, Karen" uniqKey="Fort K" first="Karën" last="Fort">Karën Fort</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-150772" status="VALID"><idno type="RNSR">201120979K</idno><orgName>Semantic Analysis of Natural Language</orgName><orgName type="acronym">SEMAGRAMME</orgName><desc><address><country key="FR"></country></address><ref type="url">http://www.inria.fr/equipes/semagramme</ref></desc><listRelation><relation active="#struct-129671" type="direct"></relation><relation active="#struct-300009" type="indirect"></relation><relation active="#struct-423086" type="direct"></relation><relation active="#struct-206040" type="indirect"></relation><relation active="#struct-413289" type="indirect"></relation><relation name="UMR7503" active="#struct-441569" type="indirect"></relation></listRelation><tutelles><tutelle active="#struct-129671" type="direct"><org type="laboratory" xml:id="struct-129671" status="VALID"><idno type="RNSR">198618246Y</idno><orgName>INRIA Nancy - Grand Est</orgName><desc><address><addrLine>615 rue du Jardin Botanique 54600 Villers-lès-Nancy</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/nancy</ref></desc><listRelation><relation active="#struct-300009" type="direct"></relation></listRelation></org></tutelle><tutelle active="#struct-300009" type="indirect"><org type="institution" xml:id="struct-300009" status="VALID"><orgName>Institut National de Recherche en Informatique et en Automatique</orgName><orgName type="acronym">Inria</orgName><desc><address><addrLine>Domaine de VoluceauRocquencourt - BP 10578153 Le Chesnay Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/en/</ref></desc></org></tutelle><tutelle active="#struct-423086" type="direct"><org type="department" xml:id="struct-423086" status="VALID"><orgName>Department of Natural Language Processing & Knowledge Discovery</orgName><orgName type="acronym">LORIA - NLPKD</orgName><desc><address><country key="FR"></country></address><ref type="url">http://www.loria.fr/la-recherche-en/departements/Knowledge-and-Language-Management</ref></desc><listRelation><relation active="#struct-206040" type="direct"></relation><relation active="#struct-300009" type="indirect"></relation><relation active="#struct-413289" type="indirect"></relation><relation name="UMR7503" active="#struct-441569" type="indirect"></relation></listRelation></org></tutelle><tutelle active="#struct-206040" type="indirect"><org type="laboratory" xml:id="struct-206040" status="VALID"><idno type="IdRef">067077927</idno><idno type="RNSR">198912571S</idno><idno type="IdUnivLorraine">[UL]RSI--</idno><orgName>Laboratoire Lorrain de Recherche en Informatique et ses Applications</orgName><orgName type="acronym">LORIA</orgName><date type="start">2012-01-01</date><desc><address><addrLine>Campus Scientifique BP 239 54506 Vandoeuvre-lès-Nancy Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.loria.fr</ref></desc><listRelation><relation active="#struct-300009" type="direct"></relation><relation active="#struct-413289" type="direct"></relation><relation name="UMR7503" active="#struct-441569" type="direct"></relation></listRelation></org></tutelle><tutelle active="#struct-413289" type="indirect"><org type="institution" xml:id="struct-413289" status="VALID"><idno type="IdRef">157040569</idno><idno type="IdUnivLorraine">[UL]100--</idno><orgName>Université de Lorraine</orgName><orgName type="acronym">UL</orgName><date type="start">2012-01-01</date><desc><address><addrLine>34 cours Léopold - CS 25233 - 54052 Nancy cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.univ-lorraine.fr/</ref></desc></org></tutelle><tutelle name="UMR7503" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Nancy</settlement><settlement type="city">Metz</settlement><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region></placeName><orgName type="university">Université de Lorraine</orgName></affiliation></author><author><name sortKey="Francois, Claire" sort="Francois, Claire" uniqKey="Francois C" first="Claire" last="François">Claire François</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-1814" status="VALID"><orgName>Institut de l'information scientifique et technique</orgName><orgName type="acronym">INIST</orgName><desc><address><addrLine>2, Allée du Parc de Brabois CS 10310 F-54519 Vandoeuvre-lès-Nancy</addrLine><country key="FR"></country></address><ref type="url">http://www.inist.fr</ref></desc><listRelation><relation name="UPS76" active="#struct-441569" type="direct"></relation></listRelation><tutelles><tutelle name="UPS76" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Galibert, Olivier" sort="Galibert, Olivier" uniqKey="Galibert O" first="Olivier" last="Galibert">Olivier Galibert</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-34522" status="VALID"><orgName>Laboratoire National de Métrologie et d'Essais - Institut National de Métrologie</orgName><orgName type="acronym">LNE- INM</orgName><desc><address><addrLine>61, rue du Landy- 93210 La plaine Saint Denis</addrLine><country key="FR"></country></address><ref type="url">http://www.cnam.fr/inm</ref></desc><listRelation><relation active="#struct-303634" type="direct"></relation></listRelation><tutelles><tutelle active="#struct-303634" type="direct"><org type="institution" xml:id="struct-303634" status="VALID"><orgName>Conservatoire National des Arts et Métiers [CNAM] : EA2367</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Grouin, Cyril" sort="Grouin, Cyril" uniqKey="Grouin C" first="Cyril" last="Grouin">Cyril Grouin</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-202" status="OLD"><orgName>Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]</orgName><orgName type="acronym">LIMSI</orgName><desc><address><addrLine>Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX</addrLine><country key="FR"></country></address><ref type="url">http://www.limsi.fr/</ref></desc><listRelation><relation name="UPR3251" active="#struct-441569" type="direct"></relation><relation active="#struct-92966" type="direct"></relation><relation active="#struct-93591" type="direct"></relation></listRelation><tutelles><tutelle name="UPR3251" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-92966" type="direct"><org type="institution" xml:id="struct-92966" status="VALID"><orgName>Université Paris-Sud - Paris 11</orgName><orgName type="acronym">UP11</orgName><desc><address><addrLine>Bâtiment 300 - 91405 Orsay cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.u-psud.fr/</ref></desc></org></tutelle><tutelle active="#struct-93591" type="direct"><org type="institution" xml:id="struct-93591" status="VALID"><orgName>Université Pierre et Marie Curie - Paris 6</orgName><orgName type="acronym">UPMC</orgName><desc><address><addrLine>4 place Jussieu - 75005 Paris</addrLine><country key="FR"></country></address><ref type="url">http://www.upmc.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Kahn, Juliette" sort="Kahn, Juliette" uniqKey="Kahn J" first="Juliette" last="Kahn">Juliette Kahn</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-34522" status="VALID"><orgName>Laboratoire National de Métrologie et d'Essais - Institut National de Métrologie</orgName><orgName type="acronym">LNE- INM</orgName><desc><address><addrLine>61, rue du Landy- 93210 La plaine Saint Denis</addrLine><country key="FR"></country></address><ref type="url">http://www.cnam.fr/inm</ref></desc><listRelation><relation active="#struct-303634" type="direct"></relation></listRelation><tutelles><tutelle active="#struct-303634" type="direct"><org type="institution" xml:id="struct-303634" status="VALID"><orgName>Conservatoire National des Arts et Métiers [CNAM] : EA2367</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Rosset, Sophie" sort="Rosset, Sophie" uniqKey="Rosset S" first="Sophie" last="Rosset">Sophie Rosset</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-202" status="OLD"><orgName>Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]</orgName><orgName type="acronym">LIMSI</orgName><desc><address><addrLine>Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX</addrLine><country key="FR"></country></address><ref type="url">http://www.limsi.fr/</ref></desc><listRelation><relation name="UPR3251" active="#struct-441569" type="direct"></relation><relation active="#struct-92966" type="direct"></relation><relation active="#struct-93591" type="direct"></relation></listRelation><tutelles><tutelle name="UPR3251" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-92966" type="direct"><org type="institution" xml:id="struct-92966" status="VALID"><orgName>Université Paris-Sud - Paris 11</orgName><orgName type="acronym">UP11</orgName><desc><address><addrLine>Bâtiment 300 - 91405 Orsay cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.u-psud.fr/</ref></desc></org></tutelle><tutelle active="#struct-93591" type="direct"><org type="institution" xml:id="struct-93591" status="VALID"><orgName>Université Pierre et Marie Curie - Paris 6</orgName><orgName type="acronym">UPMC</orgName><desc><address><addrLine>4 place Jussieu - 75005 Paris</addrLine><country key="FR"></country></address><ref type="url">http://www.upmc.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Zweigenbaum, Pierre" sort="Zweigenbaum, Pierre" uniqKey="Zweigenbaum P" first="Pierre" last="Zweigenbaum">Pierre Zweigenbaum</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-202" status="OLD"><orgName>Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]</orgName><orgName type="acronym">LIMSI</orgName><desc><address><addrLine>Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX</addrLine><country key="FR"></country></address><ref type="url">http://www.limsi.fr/</ref></desc><listRelation><relation name="UPR3251" active="#struct-441569" type="direct"></relation><relation active="#struct-92966" type="direct"></relation><relation active="#struct-93591" type="direct"></relation></listRelation><tutelles><tutelle name="UPR3251" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-92966" type="direct"><org type="institution" xml:id="struct-92966" status="VALID"><orgName>Université Paris-Sud - Paris 11</orgName><orgName type="acronym">UP11</orgName><desc><address><addrLine>Bâtiment 300 - 91405 Orsay cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.u-psud.fr/</ref></desc></org></tutelle><tutelle active="#struct-93591" type="direct"><org type="institution" xml:id="struct-93591" status="VALID"><orgName>Université Pierre et Marie Curie - Paris 6</orgName><orgName type="acronym">UPMC</orgName><desc><address><addrLine>4 place Jussieu - 75005 Paris</addrLine><country key="FR"></country></address><ref type="url">http://www.upmc.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">HAL</idno><idno type="RBID">Hal:hal-00769639</idno><idno type="halId">hal-00769639</idno><idno type="halUri">https://hal.archives-ouvertes.fr/hal-00769639</idno><idno type="url">https://hal.archives-ouvertes.fr/hal-00769639</idno><date when="2012-12-10">2012-12-10</date><idno type="wicri:Area/Hal/Corpus">003084</idno><idno type="wicri:Area/Hal/Curation">003084</idno><idno type="wicri:Area/Hal/Checkpoint">001423</idno><idno type="wicri:explorRef" wicri:stream="Hal" wicri:step="Checkpoint">001423</idno><idno type="wicri:Area/Main/Merge">001839</idno><idno type="wicri:Area/Main/Curation">001811</idno><idno type="wicri:Area/Main/Exploration">001811</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en">Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics</title><author><name sortKey="Mathet, Yann" sort="Mathet, Yann" uniqKey="Mathet Y" first="Yann" last="Mathet">Yann Mathet</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-388300" status="VALID"><orgName>Equipe Hultech - Laboratoire GREYC - UMR6072</orgName><desc><address><country key="FR"></country></address></desc><listRelation><relation active="#struct-150" type="direct"></relation><relation name="UMR6072" active="#struct-441569" type="indirect"></relation><relation active="#struct-300358" type="indirect"></relation><relation active="#struct-300266" type="indirect"></relation></listRelation><tutelles><tutelle active="#struct-150" type="direct"><org type="laboratory" xml:id="struct-150" status="VALID"><orgName>Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen</orgName><orgName type="acronym">GREYC</orgName><desc><address><addrLine>Boulevard du Maréchal Juin - 14050 CAEN Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.greyc.fr</ref></desc><listRelation><relation name="UMR6072" active="#struct-441569" type="direct"></relation><relation active="#struct-300358" type="direct"></relation><relation active="#struct-300266" type="direct"></relation></listRelation></org></tutelle><tutelle name="UMR6072" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-300358" type="indirect"><org type="institution" xml:id="struct-300358" status="VALID"><orgName>Ecole Nationale Supérieure d'Ingénieurs de Caen</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300266" type="indirect"><org type="institution" xml:id="struct-300266" status="INCOMING"><orgName>Université de Caen Basse-Normandie</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Caen</settlement><region type="region" nuts="2">Région Normandie</region><region type="region" nuts="2">Basse-Normandie</region></placeName><orgName type="university">Université de Caen Basse-Normandie</orgName></affiliation></author><author><name sortKey="Widlocher, Antoine" sort="Widlocher, Antoine" uniqKey="Widlocher A" first="Antoine" last="Widlöcher">Antoine Widlöcher</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-388594" status="VALID"><orgName>Equipe CODAG - Laboratoire GREYC - UMR6072</orgName><desc><address><country key="FR"></country></address></desc><listRelation><relation active="#struct-150" type="direct"></relation><relation name="UMR6072" active="#struct-441569" type="indirect"></relation><relation active="#struct-300358" type="indirect"></relation><relation active="#struct-300266" type="indirect"></relation></listRelation><tutelles><tutelle active="#struct-150" type="direct"><org type="laboratory" xml:id="struct-150" status="VALID"><orgName>Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen</orgName><orgName type="acronym">GREYC</orgName><desc><address><addrLine>Boulevard du Maréchal Juin - 14050 CAEN Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.greyc.fr</ref></desc><listRelation><relation name="UMR6072" active="#struct-441569" type="direct"></relation><relation active="#struct-300358" type="direct"></relation><relation active="#struct-300266" type="direct"></relation></listRelation></org></tutelle><tutelle name="UMR6072" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-300358" type="indirect"><org type="institution" xml:id="struct-300358" status="VALID"><orgName>Ecole Nationale Supérieure d'Ingénieurs de Caen</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300266" type="indirect"><org type="institution" xml:id="struct-300266" status="INCOMING"><orgName>Université de Caen Basse-Normandie</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Caen</settlement><region type="region" nuts="2">Région Normandie</region><region type="region" nuts="2">Basse-Normandie</region></placeName><orgName type="university">Université de Caen Basse-Normandie</orgName></affiliation></author><author><name sortKey="Fort, Karen" sort="Fort, Karen" uniqKey="Fort K" first="Karën" last="Fort">Karën Fort</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-150772" status="VALID"><idno type="RNSR">201120979K</idno><orgName>Semantic Analysis of Natural Language</orgName><orgName type="acronym">SEMAGRAMME</orgName><desc><address><country key="FR"></country></address><ref type="url">http://www.inria.fr/equipes/semagramme</ref></desc><listRelation><relation active="#struct-129671" type="direct"></relation><relation active="#struct-300009" type="indirect"></relation><relation active="#struct-423086" type="direct"></relation><relation active="#struct-206040" type="indirect"></relation><relation active="#struct-413289" type="indirect"></relation><relation name="UMR7503" active="#struct-441569" type="indirect"></relation></listRelation><tutelles><tutelle active="#struct-129671" type="direct"><org type="laboratory" xml:id="struct-129671" status="VALID"><idno type="RNSR">198618246Y</idno><orgName>INRIA Nancy - Grand Est</orgName><desc><address><addrLine>615 rue du Jardin Botanique 54600 Villers-lès-Nancy</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/nancy</ref></desc><listRelation><relation active="#struct-300009" type="direct"></relation></listRelation></org></tutelle><tutelle active="#struct-300009" type="indirect"><org type="institution" xml:id="struct-300009" status="VALID"><orgName>Institut National de Recherche en Informatique et en Automatique</orgName><orgName type="acronym">Inria</orgName><desc><address><addrLine>Domaine de VoluceauRocquencourt - BP 10578153 Le Chesnay Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/en/</ref></desc></org></tutelle><tutelle active="#struct-423086" type="direct"><org type="department" xml:id="struct-423086" status="VALID"><orgName>Department of Natural Language Processing & Knowledge Discovery</orgName><orgName type="acronym">LORIA - NLPKD</orgName><desc><address><country key="FR"></country></address><ref type="url">http://www.loria.fr/la-recherche-en/departements/Knowledge-and-Language-Management</ref></desc><listRelation><relation active="#struct-206040" type="direct"></relation><relation active="#struct-300009" type="indirect"></relation><relation active="#struct-413289" type="indirect"></relation><relation name="UMR7503" active="#struct-441569" type="indirect"></relation></listRelation></org></tutelle><tutelle active="#struct-206040" type="indirect"><org type="laboratory" xml:id="struct-206040" status="VALID"><idno type="IdRef">067077927</idno><idno type="RNSR">198912571S</idno><idno type="IdUnivLorraine">[UL]RSI--</idno><orgName>Laboratoire Lorrain de Recherche en Informatique et ses Applications</orgName><orgName type="acronym">LORIA</orgName><date type="start">2012-01-01</date><desc><address><addrLine>Campus Scientifique BP 239 54506 Vandoeuvre-lès-Nancy Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.loria.fr</ref></desc><listRelation><relation active="#struct-300009" type="direct"></relation><relation active="#struct-413289" type="direct"></relation><relation name="UMR7503" active="#struct-441569" type="direct"></relation></listRelation></org></tutelle><tutelle active="#struct-413289" type="indirect"><org type="institution" xml:id="struct-413289" status="VALID"><idno type="IdRef">157040569</idno><idno type="IdUnivLorraine">[UL]100--</idno><orgName>Université de Lorraine</orgName><orgName type="acronym">UL</orgName><date type="start">2012-01-01</date><desc><address><addrLine>34 cours Léopold - CS 25233 - 54052 Nancy cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.univ-lorraine.fr/</ref></desc></org></tutelle><tutelle name="UMR7503" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Nancy</settlement><settlement type="city">Metz</settlement><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region></placeName><orgName type="university">Université de Lorraine</orgName></affiliation></author><author><name sortKey="Francois, Claire" sort="Francois, Claire" uniqKey="Francois C" first="Claire" last="François">Claire François</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-1814" status="VALID"><orgName>Institut de l'information scientifique et technique</orgName><orgName type="acronym">INIST</orgName><desc><address><addrLine>2, Allée du Parc de Brabois CS 10310 F-54519 Vandoeuvre-lès-Nancy</addrLine><country key="FR"></country></address><ref type="url">http://www.inist.fr</ref></desc><listRelation><relation name="UPS76" active="#struct-441569" type="direct"></relation></listRelation><tutelles><tutelle name="UPS76" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Galibert, Olivier" sort="Galibert, Olivier" uniqKey="Galibert O" first="Olivier" last="Galibert">Olivier Galibert</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-34522" status="VALID"><orgName>Laboratoire National de Métrologie et d'Essais - Institut National de Métrologie</orgName><orgName type="acronym">LNE- INM</orgName><desc><address><addrLine>61, rue du Landy- 93210 La plaine Saint Denis</addrLine><country key="FR"></country></address><ref type="url">http://www.cnam.fr/inm</ref></desc><listRelation><relation active="#struct-303634" type="direct"></relation></listRelation><tutelles><tutelle active="#struct-303634" type="direct"><org type="institution" xml:id="struct-303634" status="VALID"><orgName>Conservatoire National des Arts et Métiers [CNAM] : EA2367</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Grouin, Cyril" sort="Grouin, Cyril" uniqKey="Grouin C" first="Cyril" last="Grouin">Cyril Grouin</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-202" status="OLD"><orgName>Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]</orgName><orgName type="acronym">LIMSI</orgName><desc><address><addrLine>Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX</addrLine><country key="FR"></country></address><ref type="url">http://www.limsi.fr/</ref></desc><listRelation><relation name="UPR3251" active="#struct-441569" type="direct"></relation><relation active="#struct-92966" type="direct"></relation><relation active="#struct-93591" type="direct"></relation></listRelation><tutelles><tutelle name="UPR3251" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-92966" type="direct"><org type="institution" xml:id="struct-92966" status="VALID"><orgName>Université Paris-Sud - Paris 11</orgName><orgName type="acronym">UP11</orgName><desc><address><addrLine>Bâtiment 300 - 91405 Orsay cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.u-psud.fr/</ref></desc></org></tutelle><tutelle active="#struct-93591" type="direct"><org type="institution" xml:id="struct-93591" status="VALID"><orgName>Université Pierre et Marie Curie - Paris 6</orgName><orgName type="acronym">UPMC</orgName><desc><address><addrLine>4 place Jussieu - 75005 Paris</addrLine><country key="FR"></country></address><ref type="url">http://www.upmc.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Kahn, Juliette" sort="Kahn, Juliette" uniqKey="Kahn J" first="Juliette" last="Kahn">Juliette Kahn</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-34522" status="VALID"><orgName>Laboratoire National de Métrologie et d'Essais - Institut National de Métrologie</orgName><orgName type="acronym">LNE- INM</orgName><desc><address><addrLine>61, rue du Landy- 93210 La plaine Saint Denis</addrLine><country key="FR"></country></address><ref type="url">http://www.cnam.fr/inm</ref></desc><listRelation><relation active="#struct-303634" type="direct"></relation></listRelation><tutelles><tutelle active="#struct-303634" type="direct"><org type="institution" xml:id="struct-303634" status="VALID"><orgName>Conservatoire National des Arts et Métiers [CNAM] : EA2367</orgName><desc><address><country key="FR"></country></address></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Rosset, Sophie" sort="Rosset, Sophie" uniqKey="Rosset S" first="Sophie" last="Rosset">Sophie Rosset</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-202" status="OLD"><orgName>Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]</orgName><orgName type="acronym">LIMSI</orgName><desc><address><addrLine>Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX</addrLine><country key="FR"></country></address><ref type="url">http://www.limsi.fr/</ref></desc><listRelation><relation name="UPR3251" active="#struct-441569" type="direct"></relation><relation active="#struct-92966" type="direct"></relation><relation active="#struct-93591" type="direct"></relation></listRelation><tutelles><tutelle name="UPR3251" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-92966" type="direct"><org type="institution" xml:id="struct-92966" status="VALID"><orgName>Université Paris-Sud - Paris 11</orgName><orgName type="acronym">UP11</orgName><desc><address><addrLine>Bâtiment 300 - 91405 Orsay cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.u-psud.fr/</ref></desc></org></tutelle><tutelle active="#struct-93591" type="direct"><org type="institution" xml:id="struct-93591" status="VALID"><orgName>Université Pierre et Marie Curie - Paris 6</orgName><orgName type="acronym">UPMC</orgName><desc><address><addrLine>4 place Jussieu - 75005 Paris</addrLine><country key="FR"></country></address><ref type="url">http://www.upmc.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author><author><name sortKey="Zweigenbaum, Pierre" sort="Zweigenbaum, Pierre" uniqKey="Zweigenbaum P" first="Pierre" last="Zweigenbaum">Pierre Zweigenbaum</name><affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-202" status="OLD"><orgName>Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]</orgName><orgName type="acronym">LIMSI</orgName><desc><address><addrLine>Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX</addrLine><country key="FR"></country></address><ref type="url">http://www.limsi.fr/</ref></desc><listRelation><relation name="UPR3251" active="#struct-441569" type="direct"></relation><relation active="#struct-92966" type="direct"></relation><relation active="#struct-93591" type="direct"></relation></listRelation><tutelles><tutelle name="UPR3251" active="#struct-441569" type="direct"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="IdRef">02636817X</idno><idno type="ISNI">0000000122597504</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-92966" type="direct"><org type="institution" xml:id="struct-92966" status="VALID"><orgName>Université Paris-Sud - Paris 11</orgName><orgName type="acronym">UP11</orgName><desc><address><addrLine>Bâtiment 300 - 91405 Orsay cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.u-psud.fr/</ref></desc></org></tutelle><tutelle active="#struct-93591" type="direct"><org type="institution" xml:id="struct-93591" status="VALID"><orgName>Université Pierre et Marie Curie - Paris 6</orgName><orgName type="acronym">UPMC</orgName><desc><address><addrLine>4 place Jussieu - 75005 Paris</addrLine><country key="FR"></country></address><ref type="url">http://www.upmc.fr/</ref></desc></org></tutelle></tutelles></hal:affiliation><country>France</country></affiliation></author></analytic></biblStruct></sourceDesc></fileDesc><profileDesc><textClass><keywords scheme="mix" xml:lang="en"><term>evaluation</term><term>evaluation.</term><term>inter-annotator agreement</term><term>manual corpus annotation</term></keywords></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Computing inter-annotator agreement measures on a manually annotated corpus is necessary to evaluate the reliability of its annotation. However, the interpretation of the obtained results is recognized as highly arbitrary. We describe in this article a method and a tool that we developed which "shuffles" a reference annotation according to different error paradigms, thereby creating artificial annotations with controlled errors. Agreement measures are computed on these corpora, and the obtained results are used to model the behavior of these measures and understand their actual meaning.</div></front></TEI><affiliations><list><country><li>France</li></country><region><li>Basse-Normandie</li><li>Grand Est</li><li>Lorraine (région)</li><li>Région Normandie</li></region><settlement><li>Caen</li><li>Metz</li><li>Nancy</li></settlement><orgName><li>Université de Caen Basse-Normandie</li><li>Université de Lorraine</li></orgName></list><tree><country name="France"><region name="Région Normandie"><name sortKey="Mathet, Yann" sort="Mathet, Yann" uniqKey="Mathet Y" first="Yann" last="Mathet">Yann Mathet</name></region><name sortKey="Fort, Karen" sort="Fort, Karen" uniqKey="Fort K" first="Karën" last="Fort">Karën Fort</name><name sortKey="Francois, Claire" sort="Francois, Claire" uniqKey="Francois C" first="Claire" last="François">Claire François</name><name sortKey="Galibert, Olivier" sort="Galibert, Olivier" uniqKey="Galibert O" first="Olivier" last="Galibert">Olivier Galibert</name><name sortKey="Grouin, Cyril" sort="Grouin, Cyril" uniqKey="Grouin C" first="Cyril" last="Grouin">Cyril Grouin</name><name sortKey="Kahn, Juliette" sort="Kahn, Juliette" uniqKey="Kahn J" first="Juliette" last="Kahn">Juliette Kahn</name><name sortKey="Rosset, Sophie" sort="Rosset, Sophie" uniqKey="Rosset S" first="Sophie" last="Rosset">Sophie Rosset</name><name sortKey="Widlocher, Antoine" sort="Widlocher, Antoine" uniqKey="Widlocher A" first="Antoine" last="Widlöcher">Antoine Widlöcher</name><name sortKey="Zweigenbaum, Pierre" sort="Zweigenbaum, Pierre" uniqKey="Zweigenbaum P" first="Pierre" last="Zweigenbaum">Pierre Zweigenbaum</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Lorraine/explor/InforLorV4/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 001811 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 001811 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Lorraine
|area= InforLorV4
|flux= Main
|étape= Exploration
|type= RBID
|clé= Hal:hal-00769639
|texte= Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics
}}
| This area was generated with Dilib version V0.6.33. |  |